Software development is expensive, and a large proportion of the cost is in testing and debugging, followed by support and maintenance of the delivered product. Real-time software presents a peculiar challenge to test, debug, and support, since it typically operates in an environment where it must service multiple, diverse events, occurring with unpredictable sequence and timing, within rigorous time constraints. Testing and debugging is often most easily done, and for some tests and problems may only be done, in the real-time environment in which the product is designed to operate. Certain tests or performance analyses of the software are best, or are required to be, performed while the software is fully operating in the complete system context, and under severe stresses. The most difficult to find problems may manifest themselves only in such an environment.
Complicated software is naturally divided into subsystems depending on functionality, and typically there is an “assignment” of software developers to specific subsystems, such that certain of those developers are familiar with a given subsystem, but not familiar with others. Each test, debug, or support effort may involve one or a few subsystems out of many. A special class of problems involves “wild writes.” A wild write occurs when one subsystem in the software incorrectly writes into a region of memory often unrelated to its own operation. This could cause a different and unrelated subsystem that uses the now-corrupted memory to exhibit unusual or “impossible” behavior.
Support of a product after it is delivered often requires solving problems that arise only in a particular customer's application. Often, such support, or support of later enhancements to the software, faces the added difficulty that it must be done by personnel who were not involved in the original software development and may not even be able to consult with those who were.
Additionally, the processors on which real-time software programs run are typically kept very busy performing the required processing tasks. They typically have only a small proportion of idle time available, or a small margin in certain real-time critical paths, in which to perform other desirable functions such as diagnostic services. This situation is caused by marketplace pressures to produce the lowest-cost hardware solution, together with the human tendency to optimistically underestimate the size and complexity of the final software product.
Testing, debugging, and supporting real-time software can be aided by tools that provide diagnostic information about the operation of the software. Some tools, such as in-circuit emulators, require external hardware to be connected to the system under test. These tools are often costly and time consuming to setup and use, and impractical or impossible to use on a product that is in active use at a customer's site.
Some tools require halting the operation of the software, or at least a thread of control, in order to obtain more detailed diagnostic information, such as the values of various software parameters at specific points during operation. However, the very nature of real-time software precludes the kind of halting that is required to gather the needed information in many cases.
Some tools require software to be instrumented by a utility during an automated addition to the software build process. The utility may generate files for use by other tool utilities that interpret and display the diagnostic information logged by the automatically generated instrumentation. Because at least most of the instrumentation is automatically generated, and does not appear in the software source files, interaction with the instrumentation on an operational software package at run-time (e.g., to specify events upon which the log should be started or stopped) may require using a tool utility that provides a sophisticated graphical user interface (GUI). This type of tool frees software developers from the work of instrumenting their software. There is also the option of leaving expensive diagnostic instrumentation out of the delivered product. The effort to develop and support such tools is large, and significant training is necessary to use them. They may also limit a developer's capability to add custom instrumentation to the software, and increase the effort needed to do so. Gathering diagnostic information from a particular instrumented software load may require obtaining the unique files generated for that specific software load by the instrumentation utility.
Some tools enable developers to easily add instrumentation to their software which will log the diagnostic information of interest in a user-specified format and with explanatory text, and do so in real-time. An example of such a tool would be the C/C++ “printf” utility. With a single line of instrumentation, a developer specifies the data to be logged, along with the formatting and accompanying explanatory text. After formatting the data accordingly, the resultant, formatted record is logged, either in some kind of storage directly accessible by the processor, or by transmitting the formatted record over an interface to be logged by another piece of equipment. These tools are hampered in their ability to provide much of the desired diagnostic information, for several reasons. The formatting step is relatively expensive of processor real-time compared to the amount of processor real-time to get to the next event which generates information of interest. Also, the directly accessible storage is relatively small compared with the desired volume of information. If larger storage has been provided for direct access by the processor, it not only adds to the cost of every product, but also typically has a slower interface, and takes longer to log the same information. Where the formatted records are transmitted over an interface to a separate piece of equipment to be logged, the bandwidth of the interface is much less than the desired logging rate, and the transmission takes additional processor real-time. Further, the presence in the software load of the many format specifiers themselves occupies precious, directly accessible storage space.
To make such tools more useful, some have incorporated means by which the instrumentation may be divided into various categories, wherein the individual categories can be separately enabled to log diagnostic information, while disabled instrumentation adds only the lesser overhead of a function call that does little more than test for enabled status before returning. In this way, a much larger amount of instrumentation can reside in the software, and the user chooses to enable only those categories that are most critical to providing the diagnostic information needed for a particular task. Nevertheless, much compromise is typically necessary for accommodating limitations in processor real-time, storage space, and/or interface bandwidth. If necessary, this diagnostic instrumentation can also be completely left out of the delivered product, while still leaving it in the software source code for building into versions intended for the lab only, by simply defining the instrumenting function name(s) as null macros. However, if that is done, then the familiar tools, previously relied upon, are no longer available when providing customer support and/or when trying to find any tough problems that managed to elude detection until after the product was delivered.